A Nearest-Neighbour Approach to Estimation of Entropies

The concept of Shannon entropy as a measure of disorder is introduced and the generalisations of the Rényi and Tsallis entropy are motivated and defined. A number of different estimators for Shannon, Rényi and Tsallis entropy are defined in the theoretical part and compared by simulation in the practical part. In this work the nearest neighbour estimator presented in Leonenko and Pronzato (2010) is compared to spacing based estimators presented in Beirlant et al. (1997) and Song (2000) for the Shannon entropy of one-dimensional distributions. For another special case of entropy, the quadratic entropy, the estimator given in Källberg et al. (2014) is compared with the nearest neighbour estimator for multidimensional densities. Comparisons focus on bias and variance for a given sample size and are executed with simulation studies. Based on the simulations, suggestions for which estimator to use under given conditions are derived. Depending on the conditions different estimators perform better than others; one estimator was not found to be universally superior.